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SUMMARY 


Measurements  of  aircraft  gas  turbine  engine  performance  in  the  altitude 
test  facility  at  RAE(P)  are  subject  to  a  small  amount  of  uncertainty  resulting 
from  a  combination  of  precision  (or  random)  errors  and  bias  (or  systematic) 
errors.  The  limits  of  the  precision  errors  can  be  readily  calculated  by  stat¬ 
istical  analysis  of  the  results  measured  during  the  engine  tests.  Bias  limits 
are  not  directly  observable  in  the  test  results,  but  can  be  predicted  by  a  com¬ 
prehensive  assessment  of  all  possible  sources  of  error,  which  are  propagated  to 
the  test  results. 

Many  people  find  these  methods  difficult  to  comprehend  and  apply  and  this 
Memorandum  has  been  written  for  their  benefit.  It  is  a  guide  not  only  for  engine 
test  staff  at  Pyestock  but  also  for  their  customers  who  need  to  be  assured  of  the 
rigorous  attention  given  to  identifying  and  reducing  measurement  uncertainty. 
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1  INTRODUCTION 

Engine  performance  measurements  in  the  altitude  test  facility  at  Pyestock 
are  expected  to  be  of  "good  accuracy".  This  is  a  qualitative  feeling  engendered 
by  the  knowledge  that  most  of  the  instrumentation  is  calibrated  and  the  perform¬ 
ance  results  are  scrutinised  to  catch  any  bad  points  as  they  come  off  the 
computer  during  the  engine  tests. 

From  time  to  time  a  complete  quantitative  assessment  of  engine  test 
uncertainty  (a  better  technical  word  than  "accuracy")  is  made.  This  may  be  a 
contractual  requirement  for  a  new  engine  project,  or  it  might  be  needed  for  the 
calibration  of  a  flight  engine  for  which  the  calculated  uncertainty  would  be 
propagated  to  the  aircraft  performance  in  flight.  Another  instance  might  be  for 
"trouble-shooting"  to  investigate  the  cause  of  an  anomalous  performance  result  so 
as  to  identify  the  most  important  sources  of  erroi  where  corrective  action  may  be 
applied  for  best  effect. 

This  Memorandum  sets  out  in  fairly  simple  terms  the  main  features  of  the 
Uncertainty  Methodology  used  at  Pyestock.  It  is  largely  based  on  the  methodology 
of  Abernethy  and  Thompson1,  with  some  simplification  and  change  of  emphasis  to 
suit  local  practices  and  resources.  The  methodology  has  also  been  extended  in 
one  respect  to  cover  graphical  effects  by  which  a  performance  curve  is  shifted 
due  to  bias  errors  propagated  through  the  horizontal  axis. 

Some  of  the  material  of  the  present  report  comes  from  experience  with  the 
Uniform  Engine  Test  Programme  (UETP)2'4,  in  which  RAE(P)  participated  with  other 
international  facilities  to  test  and  compare  results  from  a  pair  of  J57  engines. 

Some  of  the  phraseology  may  be  unfamiliar,  or  have  special  meanings,  but  it 
is  important  that  this  be  clearly  understood.  The  reader  is  offered  a  prelimi¬ 
nary  exposure  to  some  of  the  main  terms  in  the  Glossary,  immediately  ahead  of  the 
Notation.  A  fuller  explanation  is  given  in  the  main  text.  Subsequently,  it  is 
hoped,  the  Glossary  will  serve  as  a  convenient  aide  memoire  for  future  use. 

2  UNCERTAINTY  MODELS 

Fig  1  indicates  the  notional  model  by  which  random  (precision)  and  bias 
(systematic)  errors  occur.  In  the  mind's  eye  there  is  a  true  value  whose 
position  or  value  is  unknown.  A  fixed  tias  error,  3  ,  .ties  somewhere  within  the 
range  of  ±B,  the  bias  limits  which  are  centred  on  the  unknown  true  value.  The 
bias  interval  is  2E. 
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BIAS  VALUE  BIAS 

LIMIT  (UNKNOWN)  LIMIT 

i  BIAS  INTERVAL  =2B 


Fig  1  Notional  model  of  random  and  bias  errors 

Random  errors,  such  as  E  ,  are  scattered  around  the  bias  error,  following 
a  'Normal'  or  'Gaussian'  distribution.  The  experimental  standard  deviation,  s  , 
of  this  distribution  can  be  calculated  from  a  sample  of  n  points: 

V£(x  - 


Thence  the  random  uncertainty  Ur  (also  known  as  the  95%  confidence 
limits) ,  is  given  as: 


±  Ur  =  ±  t9cs  / 


(2) 


where  t9t  is  Student's  "t",  given  by  statistical  tables.  It  is  expected  that 
95%  of  the  data  points  will  fall  within  the  limits  ±  u; 
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The  notation  model  is  only  used  to  establish  the  concept.  Since  the  true 
value  is  always  unknown,  it  is  necessary  to  work  with  two  practical  models, 
described  below. 


ADDITIVE  MODEL  UADQ  =  (B  +  ts) 


ROOT  SUM  SQUARES  MODEL  URSS  = /b2  +  ( ts  )2 


Fig  2  Practical  uncertainty  models 

Fig  2  illustrates  two  practical  uncertainty  models  which  can  be  used.  Bet!", 
of  these  are  centred  on  the  observed  mean  value,  which  is  always  known,  unlike 
the  true  value.  With  the  'additive  model',  viz: 

±Uadd  -  ±  <B  +  ts>  ,  <-'! 

the  bias  limits  ±  B  are  simply  added  to  the  outer  ends  of  the  random  uncer¬ 
tainty  limits  ±  ts  . 

The  ' root -sum-squares '  model  is: 

±  u r £ “  t  Vb-'  *  (ts)  ‘  .  ^ 
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This  model  is  more  difficult  to  visualise,  but  it  implies  a  probability 
distribution  for  bias  errors  as  well  as  for  random  errors  which  are  combined  as 
independent  components.  For  further  discussion  see  Ref  5. 

Abernethy  has  shown  by  Monte  Carlo  computer  simulations,  reported  in 
Appendix  D  of  Ref  1,  and  confirmed  on  the  Pyestock  computer,  that  ±  Ua!jd  usually 
produces  99%  coverage  of  experimental  points,  whereas  ±Urss  produces  95% 
coverage . 

There  has  traditionally  been  a  preference  for  Urss  in  the  UK  and  Europe, 
but  a  preference  for  Uadd  in  the  USA.  Either,  or  both,  models  can  be  used 
because  an  important  feature  of  the  prediction  synthesis  methodology  is  that  B 
and  (ts)  components  must  be  kept  and  reported  separately. 

The  above  discussion  on  bias  and  precision  errors  relates  to  a  general 
measurement,  x  ,  illustrated  conventionally  in  Fig  1  as  a  value  on  the  horizon¬ 
tal  axis.  Similar  considerations  apply  to  graphical  values  of  y  3uch  as  SFC 
plotted  against  a  correlating  result,  z  ,  such  as  FN  as  illustrated  in  Fig  3. 
Here,  the  precision  errors  are  taken  to  be  the  values  in  the  y-direction 
scattered  about  the  curve  fit.  The  residual  standard  deviation,  RSD  takes  the 
place  of  the  precision  index,  s  .  The  curve  fit  (CF)  value  of  y  supersedes  the 
mean  value,  x  ,  and  the  separation  of  CF  from  the  true  value  (TV)  is  the  bias 
error,  p  .  As  in  Fig  1,  the  'notional  distribution’  of  bias  errors  is  centred 
on  the  true  value,  with  the  curve  fit  expected  to  lie  somewhere  between  the 
limits  (TV  +  B)  and  (TV  -  B) .  But,  since  the  true  value  is  unknown,  the 
■practical  distribution'  of  bias  errors  is  centred  on  the  curve  fit,  with  the 
true  value  expected  to  lie  somewhere  within  the  limits  (CF  +  B)  and  (CF  -  B) . 

The  statistical  properties  of  a  mean  value  and  of  a  curve  fit  are  explained 
in  section  5. 

3  THE  DEFINED  TEST  RESULT  AND  MEASUREMENT  PROCESS 

The  behaviour  of  errors  in  real  life  is  very  complicated,  and  so  it  is 
necessary  to  set  up  simplified  models  before  any  calculations  can  be  made.  In 
the  past,  many  different  models  have  been  used,  which  has  created  great  con¬ 
fusion.  An  ingenious  way  out  of  the  difficulty  is  provided  by  an  important 
feature  of  the  Abernethy  methodology1,  known  as  the  'Defined  Measurement  Process' 
(DMP) .  This  was  adopted  by  RAE(P)  in  the  early  1980s  to  enable  the  previous 
3-class  system  of  MIDAP6  to  be  converted  into  the  simpler  2-class  system,  of 
Abernethy  ■  . 


m  S 


DISTRIBUTION  OF  PRECISION  ERRORS 
(OBSERVED  SCATTER  OF  POINTS) 


(  CF+  t.RSD) 
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To  apply  the  DMP,  it  is  first  necessary  to  declare  exactly  what  is  the 
'Defined  Test  Result'  of  the  measurements.  Then  any  error  which  appears  as 
visible  scatter  in  the  defined  test  result  is  classified  as  'Precision',  and  any 
error  which  remains  constant  in  the  defined  result  is  classified  as  'Bias’.  The 
usual  practice  at  RAE(P)  is  to  declare  the  defined  result  to  be  a  single  perform¬ 
ance  curve,  such  as  SFC  vs  FN,  obtained  in  a  single  run  in  the  test  cell.  Hence 
the  scatter  about  the  curve  is  regarded  as  indicative  of  precision  errors,  while 
the  hidden  constant  errors  which  displace  the  curve  away  from  the  true  curve  are 
regarded  as  bias  errors. 

The  way  in  which  errors,  from  every  possible  source,  propagate  through  the 
whole  of  the  measurement  system,  is  briefly  described  in  the  following  sections. 
It  is  often  found  that  the  errors  which  behave  as  precision  errors  at  some  stage 
of  the  measurement  process,  end  up  as  fixed  errors  in  the  defined  result  -  this 
is  known  as  'error  f ossilisation '  because  the  previously-live  errors  have  died 
out  to  leave  fixed,  or  dead,  bias  errors  in  the  result.  An  example  of  this  is 
the  random  scatter  which  blurs  the  position  of  an  instrument  calibration  curve. 
Once  such  a  calibration  curve  is  accepted,  the  difference  between  it  and  the  true 
curve  becomes  a  fi^ed  error,  ie  bias. 

In  the  UETP  exercise*5"4  some  of  the  other  participating  facilities  adopted 
different  DMPs  from  that  used  by  RAE(P),  and  so  their  classifications  of 
precision  and  bias  were  different  from  those  of  RAE(P).  For  example,  AEPT's 
precision  class  embraced  errors  which  showed  random  variation  at  any  time 
throughout  the  whole  of  the  UETP  exercise  in  their  facility,  including  the  links 
with  their  National  Standards.  One  advantage  of  the  RAE(P)  DMP  is  that  precision 
errors,  in  the  form  of  observed  scatter  about  the  performance  curves,  are  com¬ 
pletely  amenable  to  standard  statistical  treatment  ir  the  form  of  regression 
analysis,  which  can  be  done  on-line  while  the  engine  is  running.  This  accounts 
for  all  the  precision  errors  that  actually  exist  without  the  possibility  cf  any 
of  them  being  overlooked.  Another  advantage  is  that  the  theory  of  the  ’curve 
shift  effect’,  explained  in  section  7,  strictly  only  applies  to  the  RAE(F) 
definition  of  bia«*. 

4  DIRECTION  OF  APPROACH 

The  techniques  of  'Actual  Results  Analysis'  and  'Prediction  Synthesis'  are 
alternative  directions  of  approach,  as  indicated  in  Fig  4. 

'Actual  Results  Analysis’  is  an  application  of  statistical  methods  as  car. 
be  found  in  standard  text  books.  Ref  7  is  a  popular  introduction,  Ref  8  is  r.:  r- 
rigorous  but  still  readable,  while  Ref  ?  contains  details  of  the  regression 
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ACTUAL  RESULTS  ANALYSIS 


ENGINE  TEST  RESULTS 
(Observed  Scatter 
of  Actual  Data) 


t 

Input 


I 


Random  Error  Limit  of  Curve  Fits 
(RELCF)  eg  SFC  curve 

(  Can  not  be  applied  to  Bias  Errors) 


PREDICTION  SYNTHESIS 


ESTIMATED  ELEMENTAL  ERRORS 

1.  Bias  Limits 

2.  Precision  Indices 


f 

Input 


STATISTICAL  ANALYSIS 

ERROR  PROPAGATION 

Output 

Output 

1 


Bias  Limits  of  Test  Results,  eg .  SFC 
2  Precision  Indices  of  Test  Results,  eg.  SFC 


Fig  4  Alternative  directions  of  approach  for  uncertainty  assessment 

analysis  used  to  establish  the  random  uncertainty  of  performance  curve  fit'. 
Actual  Results  Analysis  takes  the  evidence  of  data  which  actually  occurs  lr.  the 
engine  test  to  establish  the  Random  Error  Limit  of  Curve  Fit  (RELCF) .  This  is 
that  band  within  which  the  true  position  of  a  performance  curve  would  lie  if 
there  were  no  bias  errors.  A  fuller  description  is  giver,  in  section  5. 

The  alternative  direction  cf  approach  is  kncwr.  as  'Prediction  Synthesis', 
which  deals  mainly  with  bias  errors,  tut  precision  errors  car.  alsc  be  include  u 
The  methodology  cf  Abernethy  and  Thcr.psor. •  is  applied,  with  seme  slight  modifica¬ 
tions  and  changes  of  emphasis,  to  make  uncertainty  predictions.  It  is  a  prices: 
of  synthesis  because  the  complete  answer  is  built  up  (ie  synthesised)  free 
estimates  of  the  basic  elements  of  uncertainty.  A  fuller  description  is  giver.  :: 
section  6. 
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5  STATISTICAL  ANALYSIS  OF  ACTUAL  RESULTS 

5 . 1  Properties  of  mean  values  end  curve  fits 

A  fundamental  property  of  statistics  (see  Refs  7  to  9)  is  that  the  position 
of  a  mean  value,  x  ,  of  'n'  Normally  distributed  points  is  established  more 
tightly  than  a  single  point,  x  .  Its  experimental  standard  deviation  is: 


s  lx) 


1 

ifn 


s  (x) 


(5) 


This  is  indicated  in  the  following  sketch 


Fig  5  Probability  distribution  of  a  mean  value 

Equation  (5)  applies  strictly  to  Normal  lie  Gaussian)  distributions. 
Fortunately,  engine  test  data  are  usually  close  approximations  to  this  -  the 
Normality  can  be  checked  by  statistical  tests1'-  if  there  is  any  doubt. 

Engine  test  performance  results  are  usually  presented  graphically  in  the 
form  of  one  result,  y  (eg  SFC)  plotted  against  a  correlating  result,  z  leg 
FN) ,  and  a  curve  y  is  fitted  by  the  method  of  least  squares  (see  Refs  7  tc  9) . 
Sc  we  need  tc  deal  witr.  a  curve  fit,  y  instead  of  a  mean  value,  x  as 
indicated  ir.  the  following  Fig  £.  N:te  that  values  along  the  horizontal  axis 
(abscissa)  as  denoted  fcy  the  sy.-b ;  1  rather  than  'x‘  because  x.  is  used 

for  the  input  parameters,  the  errors  cf  which  are  propagated  tc  both  axes,  y 
ar.j  z  . 


95%  confidence  limits  of  y  ,  in  other  words  the  Random  Error  Limits  of  Curve  Fit 
(RELCF)  are  given  by: 


±  RELCF  -  ±  1 95 s (y) 


(7) 


These  are  shown  by  the  inner  dashed  lines  in  Fig  6.  They  are  quite  close  to  the 
fitted  curve  in  the  centre  of  the  graph,  but  they  fan  out  at  the  ends  of  the 
data,  becoming  wide  apart  for  the  extrapolations.  This  latter  effect  is  a  useful 
safety  feature. 

Fig  7  shows  a  typical  SFC  curve  fit  from  the  UETP  tests  in  Cell  3  at 
Pyestock.  In  this  example  the  best  fit  was  a  quadratic  curve,  but  sometimes  a 
cubic  curve  fit  is  justified.  At  the  SPEC  2  value  of  19.75  kN  for  FN,  the  curve 
fit  value  is  y  *  29.527  g  kN-1  s_1  with  the  RELCF  values  ±0.25%. 


28  L 


i 


SPEC  Z 


.i9?5eeE2 


26 _ 

'C  15  20  25 

FNRD  =  T  - > 


Fig  7  Typical  SFC  ct'rve  fit  with  RELCF  and  RELIP  lines 
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For  special  'Contract  Tests',  which  must  be  agreed  between  all  parties, 
these  procedures  can  be  incorporated  in  an  Engine  Test  Code10.  In  this  case  the 
above  analysis  is  developed  so  as  to  control  the  on-line  testing  in  order  to 
guarantee  that  an  agreed  random  uncertainty  is  actually  achieved.  Moreover,  this 
is  done  with  the  minimum  testing  to  achieve  that  goal,  by  continuous  computer 
analysis  during  the  testing,  taking  just  enough  points  (n)  to  ensure  the  required 
RELCF. 

5 . 2  Random  scatter  and  outlier  detection 

The  statistical  properties  of  mean  values  and  curve  fits  described  in  the 
preceding  section  5.1  assume  that  the  surrounding  scatter  comes  from  Gaussian 
distributions.  The  diagrams  of  Figs  5  and  6  are  very  much  idealised,  as  though 
these  random  errors  were  distributed  smoothly.  However,  the  real  life  behaviour 
of  random  errors  is  much  more  'lumpy'  than  this.  Even  if  there  exists  a  well- 
defined  background  Gaussian  distribution,  it  will  be  found  that  random  samples 
taken  from  it  are  always  somewhat  irregular,  especially  with  samples  of  small 
size.  Appendix  A  shows  the  sort  of  experimental  distributions  that  might  arise 
with  such  random  samples,  from  which  mean  values  and  curve  fits  are  calculated. 

In  addition  to  the  ragged  nature  of  the  actual  values  that  might  genuinely 
come  from  Gaussian  distributions,  there  are  sometimes  a  few  'outliers'  super¬ 
imposed  that  do  not  properly  belong  to  the  bulk  of  the  data.  It  is  important 
that  these  outliers  be  detected  and  eliminated  from  the  mean  values  and  curve 
fits.  Procedures  for  doing  this  are  fully  described  in  Appendix  B. 

It  is  usual  for  the  Grubbs  tests  to  be  incorporated  in  the  computer  data 
acquisition  routines  to  automatically  detect  and  eliminate  outliers  from  the 
great  number  of  mean  values  that  are  taken  for  a  typical  test  point  at  Pyestock. 

It  is  not  the  current  practice  to  apply  computer  routines  to  detect  out¬ 
liers  in  the  scatter  about  the  engine  performance  curve  fits,  but  a  possible 
procedure  is  explained  in  section  B.2.?  of  Appendix  B,  whereby  this  might  be  dene 
in  the  future. 

The  current  practice  is,  briefly,  as  follows.  If  the  position  of  the  curve 
were  known  exactly,  then  the  95%  confidence  limits  of  the  individual  points  about 
it  would  be  given  by: 

±CL9i(y)  >=  ±  t^RSD  .  (6) 


TV  F 


:  i  9 
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A  more  rigorous  expression,  which  takes  into  account  the  random  uncertainty 
of  the  curve,  is  given  by  the  Random  Error  Limit  of  Individual  Points: 

±  RELIP  -  t  [(t95RSD)2  +  RELCF2]  14  (9) 

and  these  lines  are  shown  on  Figs  6  and  7.  Since  RELCF  decreases  progressively 
as  the  number  of  points  n  increases,  it  follows  that 

RELIP  ->  1 95RSD  .  (10) 

Even  with  a  modest  number  of  points  (say  n  m  9  as  for  the  curve  fit  in 
Fig  7)  it  will  be  found  that  the  value  of  RELIP  is  quite  close  to  tgsRSD  .  With 
the  actual  data  shown  in  Fig  7,  the  RELCF  is  0.25%,  the  RSD  is  0.20%  and 
is  2.45.  Hence  from  equation  (9) 

±  RELIP  «  ±  [(2.45  X  0 . 2  0 )  2  +  0.252  ]V!  -  ±0.55% 

and  from  equation  (10) 


±  RELIP  ±  2.45  X  0.20  ‘  0.49% 

which  is  quite  a  close  approximation. 

To  complete  the  Pyestock  procedure  for  outlier  detection,  we  would  expect 
and  accept  1  point  in  20  to  fall  just  outside  the  ± RELIP  boundaries,  but  any 
point  much  further  away  from  the  curve  would  be  regarded  as  an  outlier  and  hence 
a  candidate  for  deletion.  If  deleted,  the  curve  fit  would  then  be  recalculated 
with  one  less  data  point. 

If  this  analysis  is  done  on-line  during  the  engine  test,  the  opportunity 
exists  for  extra  points  to  be  taken  to  compare  with  the  suspect  point,  to  help 
the  decision  whether  to  reject  or  not.  This  procedure  follows  the  principles  c£ 
the  Test  Code10  mentioned  at  the  end  of  section  5.1  above. 

6  PREDICTION  SYNTHESIS 

£ . 1  Preliminary  sensitivity  survey 

It  is  usual  to  start  with  a  preliminary  'sensitivity  survey'  to  get  seme 
idea  of  the  sensitivity  of  the  various  performance  results  (R*)  such  as  SFC,  te 
the  various  'basic  measurements'  (x.)  such  as  air  meter  static  pressure  (PSA). 

An  alternative  name  for  x_  is  'input  parameter',  because  it  is  an  input  tc  the 


15 


engine  performance  calculations .  This  requires  the  calculation  of  the  'influence 
coefficients'  6^  (sometimes  called  'sensitivity  factors').  In  dimensional 
terms : 

„  SRv 

9ki  -  sf.  <12> 


where  dR*  is  the  change  in  the  result  R*  due  to  a  change  dx*  in  an  input. 

In  principle,  it  should  be  possible  to  evaluate  0*i  by  differential  calculus, 
but  the  relationship  between  R*  and  xi  in  engine  performance  calculations  is 
usually  too  complicated  for  this.  Instead,  0*1  values  can  conveniently  be 
found  by  perturbation  of  the  computer  calculations,  providing  that  the  engine 
performance  program  has  been  written.  The  calculation  of  these  values  is 

itself  done  by  computer  program.  It  is  more  convenient  to  express  the  influence 
coefficients  in  relative  form: 


®ki 


xi  9r« 
Rk  3xj 


(13) 


In  practice,  not  only  are  different  influence  coefficients  required  for 
each  xj  ,  for  each  R*  ,  but  also  for  each  flight  condition.  Thus 


Fig  8  Matrix  of  influence  coefficients 

The  central  column  of  Fig  9  depicts  typical  influence  coefficients. 
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Fig  9  Propagation  ot  bias  limit*  Irom  Input  parameters  to  (SFCRD)  at  target  points 
(prediction  synthesis  for  J57  UETP  tests  In  Cell  3,  RAE,  Pyestock) 
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In  this  example,  taken  from  the  UETP  exercise  in  Cell  3,  the  matrix 
consisted  of  15  values  of  xj  ,  4  values  of  R*  and  10  flight  conditions  -  a 
total  of  600  different  influence  coefficients.  Perhaps  the  biggest  problem  here 
is  to  display  this  information  in  such  a  way  that  the  reader  can  taken  it  in! 

The  central  column  of  Fig  9  depicts  150  of  such  influence  coefficients  in  bar 
chart  form,  for  the  single  result  of  R*  -  SFCRD  . 

Having  established  these  influence  coefficients,  one  can  see  at  a  glance 
which  are  the  most  important  input  parameters  where  most  attention  must  be  given 
to  keep  the  errors  down.  Fig  9  shows  that  x5  -  PS1  (the  engine  inlet  static 
pressure)  is  the  most  important  item  for  Rk  m  SFCRD  .  At  Flight  Condition  9  the 
influence  coefficient  is 

0*5  ■=  -  3.5%/%  (14) 

that  is,  1%  error  in  PS1  will  create  -3.5%  error  in  SFCRD.  At  the  other  extreme, 
it  can  be  seen  from  Fig  9  that  Barometer  error  has  negligible  effect  on  this 
particular  result,  with  the  worst  influence  coefficient  at  Flight  Condition  8  of 

e*:;  -  o.i%/%  ns) 

that  is,  1%  error  in  Barometer  will  create  only  0.1%  error  in  SFCRD.  The  reason 
for  this  relative  immunity  to  error  in  Barometer  is  the  beneficial  effect  of 
error-cancellation  in  the  complicated  engine  performance  calculations  for  SFCRD. 

(The  left-hand  and  right-hand  columns  of  Fig  9  are  described  in  the  following 
section . ) 

6.2  Complete  prediction  synthesis 

The  strategy  is  laid  out  in  Fig  10.  The  basic  measurements  x;  (which 
were  encountered  in  section  6.1  as  input  parameters  for  the  Sensitivity  Survey) 
are  now  the  interface  between  Step  1  and  Step  2. 
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For  Step  1,  separate  'elemental  error'  tables  are  prepared  for  each  Xj  . 
Table  1  is  an  example,  where  X7  is  the  test  frame  load.  Error  sources  are 
assessed  under  four  categories  (j)  as  indicated  in  Fig  11. 
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Fig  11  Elemental  error  sources  (for  each  basic  measurement) 

Category  1  covers  the  calibration  hierarchy,  traceable  all  the  way  from 
the  measuring  instrument  (a  Bofors  shear  force  load  cell  in  Table  1)  to  the 
National  Standard  at  NPL.  At  each  stage  the  calibration  consists  of  comparing 
one  standard  with  another.  Fig  12  shows  how  the  original  biases  are  removed  by 
these  calibrations,  but  in  their  place  there  remains  a  combination  of  random 
uncertainty  with  a  total  standard  deviation  for  the  complete  calibration 
hierarchy  of 


V 


s.  +  s,  +  st 


(16) 


However,  the  residual  errors  from  this  calibration  hierarchy  are  propagated 
to  the  engine  test  as  a  single  fixed  error,  not  as  random  scatter,  ie  they  are 
•fossilised*  from  random  errors  into  bias  errors  within  the  limits 


tsi  ,  B2 


ts2 


etc  . 


(17) 


Hence 


VbT+B^  +  B Y  + 


(18) 


Fig  13  illustrates  how  any  one  stage  of  the  calibration  hierarchy  may  be 
analysed.  The  standard  deviation  in  this  case  would  be 


sone  stage 


VI 


(v:  -  y) ; 


(19) 


where  y^  is  the  ith  curve  fit. 


Fig  13  Statistical  analysis  of  different  calibration  lines 

In  the  case  cf  test  frame  load  in  Table  1,  the  calibration  hierarchy  is 
quite  short,  with  only  two  stages  plus  a  small  basic  uncertainty  of  the  Naticr.a 
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Standard.  These  elemental  bias  limits  are  entered  separately  in  Table  1,  because 
they  will  be  root-sum-squared  down  the  complete  table. 

Category  2  is  Data  Acquisition.  Fig  11  shows  this  conceptually  as  a 
collection  of  effects  in  the  instrumentation  system.  However,  at  Pyestock  the 
complete  effect  is  always  assessed  by  'end-to-end'  calibrations  of  the  measuring 
instrument  against  the  working  standard.  In  the  case  of  test  frame  load  in 
Table  1,  data  acquisition  errors  are  included  in  the  last  stage  of  Category  1  and 
the  middle  item  of  Category  4.  iThe  important  thing  is  that  no  item  should  be 
overlooked,  but  no  item  counted  twice.) 

Category  3  is  Data  Reduction.  In  Table  1,  an  elemental  bias  limit  cf 
20  N  is  assessed  due  to  nonlinearity  of  load  calibration. 

Category  4  in  Fig  11  was  introduced  at  Pyestock  to  cover  important  'non- 
instrumentation'  effects.  In  the  case  of  test  frame  load,  for  example,  these 
real  effects  are  manifestations  of  test  frame  mechanics  (stray  friction,  spring 
and  gravity  forces)  which  are  found  to  be  more  important  than  any  errors  in  the 
instrumentation.  (The  Bofors  shear  force  load  cell  with  its  fundamental 
calibration,  traceable  to  the  National  Standard,  is  responsible  for  very  little 
error . ) 

When  all  the  entries  for  elemental  errors  are  completed  for  a  particular 
basic  measurement,  such  as  test  frame  load  in  Table  1,  these  elemental  bias 
limits  are  combined  by  root-sum-squares.  In  the  case  of  Table  1  this  amounts  tc 
41  N. 

Step  1  is  complete  when  the  separate  elemental  error  tables  for  each  cf 
the  basic  measurements  have  been  completed. 

Step  2  (see  Fig  10)  consists  of  propagating  the  values  of  B_  from,  the 
basic  measurements  to  the  test  result,  R,  .  It  is  convenient  to  do  these 
calculations  in  separate  tables  such  as  Table  2,  one  for  each  flight  condition. 
Values  of  Bi  are  entered  in  Table  2  and  converted  to  %.  The  effect  of  each 
basic  measurement  is  then  propagated  to  each  result,  Rt  ,  by  means  of  the 
appropriate  influence  coefficient,  thus 


b-.  - 


For  example  in  Table  2,  for  the  result  cf  R, 


SFCRD  we  have 


1 
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i 

Bi(%) 

61  i  (%/%) 

Bjci  <%> 

1 

PSA 

0.06 

-0.12 

-0.01 

2 

PA 

0.25 

-0.12 

-0.03 

etc 

Thence  the  root-sum-squares  combination  for  all  the  inputs  is 

iB,  -  ±JZ  <21> 

for  the  result  RK  •  SFCRD  ,  at  Flight  Condition  1.  This  process  is  depicted  ir. 
bar  chart  form  in  Fig  9  for  the  result  R*  “  SFCRD  for  each  of  the  10  flight 
conditions . 

7  THE  CURVE  SHIFT  EFFECT 

Engine  performance  results  are  always  presented  as  graphical  correlations 
against  other  suitable  parameters.  Fcr  example,  SFC  would  be  correlated  against 
Net  Thrust.  Again,  Net  Thrust  might  be  correlated  against  Pressure  Ratio  F7/F2 
or,  alternatively,  against  ncn-dinersicr.a!  shaft  speed,  KH'ATl.  Thus  ar.  iscia-ed 
result  of  Net  Thrust  is  of  no  value  by  itself  without  consideration  of  its 
correlating  parameter. 

The  curve  shift  effect  to  be  described  in  the  present  document  is  a 
phenomenon  caused  by  bias  errors  (as  defined  at  RAE(P))  propagated  through  the 
horizontal  axis,  z  .  Precision  errors  of  z  (as  defined  at  RAE(P))  will  have 
some  effect  on  the  scatter  of  points  but,  since  this  is  likely  to  be  small,  it 
has  been  decided  not  to  describe  it  in  the  present  document,  ir.  order  tc  ccr.ren- 
trate  attention  on  the  more  important  effect  of  bias  errors. 

The  following  three  Figures  (14,  15,  161,  taker,  from,  actual  UE7F  result 

illustrate  the  problem,  while  Fig  17  displays  the  theory  to  be  explained  in  tr.-- 
text  after  it. 

Fig  14  is  an  example  of  a  comparison  between  the  performance  resjlts  fr:m 
some  UETP  tests  on  a  J57  engine  measured  ir.  several  international  facilities, 

displayed  as  conventional  giophs  of  Referred  .‘.'ct  Th - 1,  y  -  FNRD,  vs  Frets-:- 

Ratio,  z  -  F7/P2.  Unfortunately,  with  this  conventional  presentation,  the  ey- 
car,  hardly  see  the  difference  between  facilities.  However,  a  revised  display  .- 
Fig  15  shows  deviations  AFNRD  frem  a  datum,  iir.e,  plotted  against  the  same  F“  ;.. 
in  which  the  differences  between  facilities  car.  now  be  plainly  seen.  Pig  If 


A 


5  Enhanced  graphs  of  engine  performance  (AFNRD  vs  P7/P2) 
(easy-to-see  differences  between  facilities) 


Fig  17  Graphical  propagation  of  bias  errors  10  performance  curve 
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an  alternative  revised  display,  using  z  «  NHRD  instead  of  P7/P2.  Comparing 
Figs  15  and  16,  the  pattern  of  the  curves  from  the  various  facilities  is  seen  to 
be  drastically  altered  by  changing  the  correlating  parameter  from  z  -  P7/P2  to 
z  -  NHRD,  even  though  the  values  of  y  -  FNRD  are  exactly  the  same.  So,  it  is 
the  bias  errors  in  P7/P2  (on  the  one  hand)  or  the  bias  errors  in  NHRD  (on  the 
other  hand)  which  make  such  a  big  difference  to  the  performance  curves  of  FNRD. 

The  mechanism  of  the  graphical  propagation  of  errors  is  presented  in 
Fig  17,  in  terms  of  bias  limits.  The  theory  only  applies  to  errors  that  remain 
constant  in  the  performance  curve.  This  is  no  problem  with  the  DMP  adopted  by 
RAE(P)  which  regards  such  errors  as  bias.  However,  there  would  be  a  problem  if 
some  of  these  constant  errors  were  regarded  as  precision,  as  was  done  by  some 
UETP  participants. 

Errors  in  the  Basic  Measurements  (x^)  can  propagate  to  the  curve  in  three 
ways,  as  depicted  in  Fig  17.  The  most  obvious  route  [p]  is  travelled  by  errors 
that  only  affect  'y',  and  these  limits  are  calculated  by  standard  Abernethy 
methodology.  The  next  route  [q]  is  travelled  by  errors  that  only  affect  'z'. 
while  these  latter  have  no  effect  on  the  j -values  of  the  data  points,  they  do 
affect  the  y-position  OF  THE  CURVE  as  indicated  in  Fig  17. 

The  third  route  0' . 0'  is  followed  by  errors  that  affect  1 y '  and 

'z'  simultaneously.  Their  complete  effect  should  be  fully  allowed  for  as 
indicated  in  Fig  17,  BEFORE  it  is  combined  by  root-sum-squares  with  the  bias 
limits  from  route  [F|  and  |q].  This  process  yields  the  'Bias  Limit  of  Curve  Fit', 
thus 


BLCF 


y , indep 
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dy  1 ' 

dz  J 
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z. indep 


dz  ^ 


(22) 


Equation  (22)  can  often  be  simplified,  according  to  circumstances.  Thus, 
if  the  curve  slope,  dy/dz  is  small,  it  reduces  simply  to 
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ir.de  p 
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all 


(23) 


where  is  the  total  predicted  bias  limit  of  y  ,  propagated  from  all  basic 

measurements,  using  standard  Abernethy  methodology. 


29 


Again,  if  most  of  the  error  in  1  z*  are  independent  of  'y',  it  becomes 


BLCF 


2 

y, indep 


+ 


[B 


z, indep 


i 

dy  I  2 
dz  J 


(24) 


Or  again,  if  most  of  the  errors  in  '  z'  are  common  to  'y',  we  get 


BLCF 


[  B 
L  y 


-  B 

Z ,  com  dz  J 


(25) 


An  example  of  equation  (24)  is  provided  by  the  curves  of  FNRD  vs  P7Q2, 
where  errors  in  'y*  and  ' z'  are  practically  all  independent,  and  so  the  bias 
limits  accumulate  almost  entirely  by  root-sum-squares  as  illustrated  in  the  top 
of  Fig  18. 

The  special  case  of  equation  (25)  as  applied  to  the  SFC  curves  is 
illustrated  in  the  bottom  half  of  Fig  18  where  the  sense  of  the  slope  is  seen  to 
have  a  serious  effect  on  BLCF.  Errors  in  z  •=  FNRD  are  obviously  common  to 
y  -  SFCRD  »  WFRD/FNRD  via  the  variable  FNRD,  and  some  errors  from  other  sources 
are  common.  Because  of  this  special  relationship,  the  positive  limit  of  By  is 
always  associated  with  the  negative  limit  of  Bz  ,  and  vice  versa.  The  conse¬ 
quence  is  that  these  bias  limits  will  self-cancel  when  the  slop  is  negative,  but 
will  build  up  when  the  slop  is  positive. 

This  self-cancelling  effect  due  to  the  steep  negative  slope  of  the  SFC 
curves  in  the  UETF  exercise  at  low  P2  pressure  levels,  might  explain  why  the 
interfacility  spreads  of  the  SFC  curves  were  so  surprisingly  small  at  the  diffi¬ 
cult  high  altitude  Test  Condition  9. 

The  last  diagram  at  the  bottom  right  of  Fig  18,  illustrates  the  build-up  of 
the  common  errors  in  the  graphs  of  WalRD  vs  NLRD  due,  in  this  case,  to  the 
positive  slope.  The  common  error  here  is  the  temperature,  T1 .  The  group  WA1RD 
is  related  to  WAiVtI,  while  NLRD  is  related  to  NL/VtI .  Hence  a  positive  error  in 
T1  causes  a  positive  Bz>cor  and  a  negative  BViC=r  which  build  up  in  a  negative 
sense,  as  indicated  in  Fig  18.  On  the  other  hand,  a  negative  error  in  T1  causes 
a  negative  Bzco~  and  a  positive  Bycor  which  build  up  in  a  positive  sense. 

It  was  suggested  in  Ref  11  that  this  important  effect  of  common  error  in  T1  could 
be  a  possible  contributor  to  the  so-called  'Cell  Effect',  of  which  the  main 
symptom  was  an  apparent  difference  on  engine  airflow  between  ground  level  test 
beds  and  altitude  cells. 
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a)  GENERAL  CASE  WITH  INDEPENDENT  ERRORS  IN  y  AND  z 


eg 


IFNRD  V,  P7Q2I 


(POSITIVE  SLOPE) 


b)  SPECIAL  CASES  WITH  COMMON  ERROR  IN  y  AND  z 
NEGATIVE  SLOPE  f SFCRO  V.  FN~RD~]  POSITIVE  SLOPE  I  WAtRO  V.  NLROl 
BIAS  LIMITS  SELF-CANCEL  BIAS  LIMITS  BUILD  UP 


Fig  18  Illustration  of  extreme  cases  of  curve  shifts 
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8  HISTORICAL  REVIEW  OF  THE  METHODOLOGY 

The  methodology  described  in  this  Memorandum  originated  in  the  USA  about 
15  years  ago1-  At  that  time  a  rather  different  procedure,  developed  in  the  UK6, 
was  used  at  Pyestock.  This  latter  took  account  of  three  classes  of  error: 

(i)  short-term  random  errors, 

(ii)  errors  which  are  systematic  in  the  short  term,  but  random  in  the  long 
term, 

(iii)  long  term  systematic  errors. 

At  around  the  same  time,  contact  was  made  with  Abernethy  on  SAE  E33 
committee,  where  it  was  realised  that  both  methodologies  could  be  reconciled  by 
the  concept  of  'the  defined  measurement  process'.  This  allowed  the  3-class  MIDAP 
system  to  be  simplified  to  the  2-class  Abernethy  system,  and  so  the  latter  was 
introduced  at  Pyestock. 

After  several  years  experience  here,  culminating  in  the  UETP  exercise?'4  in 
which  a  J57  engine  was  tested  in  several  facilities  on  both  sides  of  the 
Atlantic,  the  most  valuable  features  have  been  consolidated.  The  procedures  of 
'prediction  synthesis'  are  recognised  as  essential  for  assessing  bias  error 
limits,  and  the  discipline  of  its  formal  framework  helps  to  ensure  that  no  source 
of  error  is  overlooked,  but  none  counted  twice.  A  particularly  useful  feature  is 
the  separate  'elemental  error  table'  (eg  Table  1)  for  each  basic  measurement,  or 
input  parameter.  These  tables,  together  with  the  error  propagation  chart  (eg 
Fig  9)  are  especially  useful  for  identifying  and  tracking  down  the  most  important 
error  sources.  On  the  other  hand,  the  effects  of  precision  errors  are  considered 
to  be  best  left  to  'actual  results  analysis',  without  the  need  for  the  extensive 
effort  involved  in  prediction  synthesis. 

Some  aspects  of  the  'curve  shift  effect'  had  been  studied  at  Pyestock  more 
than  a  decade  ago,  with  particular  regard  to  T1  error1-.  But  it  was  the  stimulus 
of  the  UETP  analysis  which  led  to  the  full  understanding  of  this  phenomenon. 

To  sum  up:  we  now  have  a  good,  practical  uncertainty  methodology  which  is 
understood  and  appreciated  for  engine  testing  in  all  NATO  facilities. 
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Appendix  A 

EXPERIMENT  WITH  RANDOM  NUMBERS  TO  DEMONSTRATE  STATISTICAL 
PROPERTIES  OF  MEAN  VALUES  AND  CURVE  FITS 

A.l  Objective  of  the  experiment 

When  a  curve  is  fitted  to  'n'  points  of  two-dimensional  data  by  the  method 
of  least  squares,  it  has  to  be  recognised  that  the  curve  is  only  an  estimate  of 
where  the  true  curve  really  is,  as  obscured  by  the  random  scatter  of  the  data. 

The  fitted  curve  is  the  best  estimate,  but  the  true  curve  might  be  anywhere 
within  an  uncertainty  band  whose  limits  can  be  calculated  by  analysis  of  the  data 
in  the  form  of  the  Random  Error  Limits  of  Curve  Fit  (±  RELCF) .  Section  A. 3 
demonstrates  how  well  such  RELCF  calculations  might  cover  a  true  curve  when  a 
sample  of  20  points  are  analysed. 

The  statistical  properties  of  a  least  squares  curve  are  analogous  to  the 
properties  of  a  mean  value  of  a  random  sample.  Since  the  latter  is  easier  to 
explain  and  understand,  this  will  be  discussed  first,  in  section  A. 2. 

In  both  sections  A. 2  and  A. 3  the  results  are  presented  in  alternative  ways. 
The  first  and  simplest  way  is  to  plot  the  theoretical  limits  about  the  true  mean, 
or  true  curve,  and  note  how  well  these  limits  enclose  the  experimental  mean,  cr 
experimental  curve.  Unfortunately,  in  practice  the  true  mean  and  true  curve  are 
usually  not  known  and  so  it  is  more  instructive  to  plot  the  experimental  limits 
about  the  experimental  mean,  or  experimental  curve,  and  note  how  well  these 
limits  enclose  the  true  mean,  or  true  curve. 

Thus  the  theory  of  the  text  books  is  put  to  the  test  with  an  actual  sample 
of  random  numbers.  In  this  experiment  we  know  exactly  what  the  true  mean  value 
and  true  curve  really  are,  but  we  do  not  know  in  advance  what  values  of  random 
numbers  will  crop  up.  When  the  numbers  do  appear,  it  is  interesting  and 
instructive  to  see  how  the  experimental  means  and  curve  fits  fall  within  their 
estimated  uncertainty  bands. 

A.  2  Statistical  properties  of  mean  values 

Random  number  generators  will  produce  a  series  of  numbers  (R)  from  a 
rectangular  distribution  between  0  and  1,  thus: 
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Fig  Ai  Rectangular  distribution  of  random  numbers 

AH  values  of  R  are  equally  likely  between  0  and  1,  with  a  true  mean  of 
0.5.  Not”~  that  the  area  under  the  curve  equals  1  (ie  total  probability  *•  1). 

To  generate  samples  from  a  Gaussian  distribution,  we  simply  take  a  sequence 
of  12  numbers  from  the  above,  add  them  together  and  subtract  the  constant  6.00 
from  the  sum: 


G 


-  6.00 


It  is  shown  in  Ref  Al  that  the  resulting  number  (G)  will  be  a  sample  from  a 
Gaussian  distribution,  with  the  true  mean  of  0  and  the  standard  deviation  of  1 . 
Again,  the  area  under  the  curve  equals  1.  (Note  that  O  •  1  only  when  12 
random  numbers  are  taken  from  the  generator.) 


Fig  A2  Gaussian  distribution  derived  from  12  random  numbers 
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For  the  present  experiment,  a  series  of  20  such  values  of  G  were  taken 
and  plotted  out  in  Fig  A3.  After  the  first  two  values  of  G  were  available, 
their  experimental  mean  was  calculated: 

-  <  G i  +  G?) 

G2  -  — ^  . 

Then  after  the  next  value,  G3  ,  arrived,  the  mean  of  3  was  calculated: 

(Gi  +  G;  +  G3) 


and  so  on  up  to  Gjg  . 

At  each  stage,  the  experimental  standard  deviation  was  also  calculated: 


This  is  an  estimate  of  the  standard  deviation,  the  true  value  of  which  we 
know  to  be  exactly  a  «  1  . 

In  the  top  left  of  Fig  A3  these  experimental  values  of  points,  G  and 
means,  G  ,  are  enclosed  by  the  theoretical  limits  drawn  about  the  true  mean  of 
zero.  Here,  the  lines  at  ±2<J  are  the  95%  confidence  limits  of  the  points,  G  , 
while  the  ±20  limits  of  mean  values  are  given  by: 


o (mean) 


0 


(points) 

Vn 


The  ±20  limits  remain  a  constant  distance  from  the  true  mean,  and  it  found 
that  one  out  of  20  points  falls  outside  these  limits,  as  expected.  The 
±20{mean)  limits  begin  quite  wide  apart  where  *n*  is  small,  but  converge  steadily 
as  'n'  increases.  All  the  experimental  means  are  seen  to  lie  inside  these 
limits,  while  we  might  have  expected  one  of  them  to  fall  outside,  although  the 
mean  of  4  is  quite  near  to  the  upper  limit. 

The  top  right  of  Fig  A3  shows  the  theoretical  probably  distribution  of  the 
points,  as  already  seen  in  the  text  above.  The  theoretical  distribution  of  the 
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mean  of  20  points  is  also  shown.  This  is  much  tighter  than  the  distribution  of 
points,  but  the  area  under  both  curves  equals  1  (ie  total  probability  -  1). 

Now,  in  a  practical  situation,  the  true  mean  (n)  and  theoretical  standard 
deviation  (a)  would  not  be  known.  Instead,  we  must  deal  with  the  experimental 
means  (G)  and  experimental  standard  deviations  (s) .  Hence  the  bottom  left  of 
Fig  A3  shows  the  experimental  95%  limits  drawn  about  the  experimental  means. 

This  time  it  is  necessary  to  use  a  value  of  Student's  "tgs"  instead  of  the 
theoretical  factor  of  2.  An  interesting  feature  is  that  the  Its  limits,  calcu¬ 
lated  just  after  the  wide  fourth  point  arrived,  are  much  wider  apart  than  the 
theoretical  ±2a  limits  -  ie  with  this  sample  of  only  n  ■  4  ,  including  the  wide 
point,  it  seemed  that  a  very  scattered  distribution  was  being  sampled.  However, 
by  the  time  that  n  -  20  point  had  been  taken,  the  ±ts  limits  had  tightened  such 
that  1  out  of  20  points  fell  outside. 

The  experimental  standard  deviation  of  the  mean  is  given  by 

s (points) 
s(mean)  ■=  — p - 

Vn 

and  the  its (mean)  limits,  drawn  about  the  experimental  means,  are  seen  to 
enclose  the  true  mean  easily. 

A  histogram  of  the  experimental  points  is  shown  at  the  bottom  right  of 
Fig  A3.  This  bears  a  rough  resemblance  to  the  theoretical  distribution  of 
points.  Noting  that  'relative  frequency'  is  the  same  thing  as  'probability',  we 
expect  the  area  of  the  histogram  to  be  exactly  1  (as  it  is)  just  as  for  the  area 
under  the  curves. 

It  is  important  to  emphasise  the  distinction  between  the  use  of  theoretical 
and  experimental  limits  of  the  mean  values.  The  former  is  a  notional  situation, 
useful  for  demonstration  purposes  where  we  can  draw  the  ±20 (mean)  limits  in 
advance  and  then  watch  the  experimental  means  come  along  and  fall  within  this 
band,  as  in  the  present  experiment.  The  latter  is  the  usual  practical  situation 
where  the  true  mean  is  not  known.  This  time  we  can  only  calculate  s(mean)  ir. 
retrospect,  and  expect  the  true  mean  to  lie  within  the  ±ts(mean)  band  drawn 
about  the  experimental  means. 

A.  3  Statistical  properties  of  a  curve  fit 

In  principle,  it  would  be  possible  to  show  how  successive  curve  fits 
evolved  as  the  same  size  (n)  built  up  from  n  -  4  (which  is  the  smallest  size 
for  a  quadratic  fit,  with  statistical  calculations)  to  n  -  20  .  This  would 
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follow  the  procedure  followed  for  the  mean  value  results  in  section  A. 2  and 
Fig  A3,  but  would  be  too  complicated  for  this  Appendix.  Instead,  let  us  suppose 
that  20  points  of  (z,y)  data  had  been  obtained,  with  a  sample  of  Gaussian  error 
the  same  as  Fig  A3  superimposed  upon  the  true  y  values.  The  results  of  a  quad¬ 
ratic  curve  fitted  by  least  square  to  these  data  are  shown  in  Fig  A4 .  In  the  top 
left  of  the  Figure,  the  theoretical  ±2c  limits  (of  individual  points)  and  the 
±2a  limits  (of  the  curve  fit)  are  drawn  about  the  true  curve.  As  for  the  mean 
value  example  in  Fig  A3,  it  is  seen  that  1  out  of  20  points  falls  outside  these 
20  limits,  as  expected,  while  the  curve  fit  values  all  lie  within  their  +20 
limits  -  which  is  a  little  better  than  expected.  Note  that  the  ±20  limits  of 
c  .rve  fit  become  wider  apart  as  we  move  towards  the  ends  of  the  curve. 

As  before,  in  a  practical  situation  the  true  curve  and  true  standard 
deviation  would  not  be  known.  And  so  it  is  necessary  to  draw  the  experimental 
limits  about  the  experimental  curve  fit.  The  theoretical  standard  deviation, 

O  ,  is  replaced  by  the  experimental  RSD  and,  instead  of  ±2o  (of  curve  fit) ,  we 
plot  its  (of  curve  fit),  ie  the  Random  Error  Limits  of  Curve  Fit,  1RELCF  .  To 
plot  the  95%  confidence  limits  of  individual  points,  the  limits  it.RSD  should 
strictly  be  comoined  with  RELCF  by  root-sum-squares,  to  give  the  Random  Error 
Limits  of  Individual  Points: 

±  RELIP  ■=  ±[<t.RSD)?  +  RELCF2]'4  . 


As  the  number  of  points,  n  ,  increases,  the  value  of  RELCF  decreases, 
while  t.RSD  stabilises.  In  fact,  with  the  value  of  n  as  high  as  20,  it  is  a 
good  approximation  to  say 


± RELIP  -  It.RSD 

and  these  limits  are  plotted  about  the  curve  fit  in  the  bottom  half  of  Fig  A4  . 
It  is  seen  that  1  out  of  20  points  falls  outside  the  it.RSD  limits,  while  the 
curve  fit  values  all  lie  within  iRELCF  . 

In  the  top  right  of  Fig  A4  are  tc  be  seen  the  smooth  theoretical  Gaussian 
probability  distributions,  centered  cn  the  true  curve.  The  distribution  of 
points  is  exactly  the  same  as  was  shown  in  the  top  right  of  Fig  A3  but,  instead 
of  the  distribution  of  mean  values,  we  now  have  the  distribution  of  curve  fits, 
evaluated  at  two  places  -  ie  near  the  middle  of  the  curve  where  it  is  tightest, 
and  at  the  ends  of  the  curve  where  it  is  widest. 
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The  histogram  of  the  experimental  distribution  of  points  is  shown  at  the 
bottom  right  of  Fig  hi,  centred  on  the  curve  fit.  This  is  the  same  as  the 
histogram  in  Fig  A3,  centred  on  the  experimental  mean  of  20  points,  with  the  area 
of  the  histogram  equal  to  1  exactly  (ie  total  probability  -  1) . 

A. 4  Concluding  remarks 

This  experiment  has  analysed  a  sample  of  20  random  numbers  from  a  Gaussian 
distribution.  At  the  beginning,  the  writer  did  not  know  what  the  numbers  would 
be  and,  if  the  experiment  were  to  be  repeated,  a  different  of  numbers  would 
appear.  In  fact,  the  values  in  Figs  A3  and  A4  should  never  occur  again. 

However,  these  numbers  do  demonstrate  the  sort  of  way  that  mean  values  and  curve 
fits  are  expected  to  occur  within  the  limits  that  can  be  calculated  from  the  data 
by  statistical  principles. 

Finally,  it  should  be  remembered  that  this  Appendix  only  deals  with  the 
properties  of  random  errors  from  a  Gaussian  distribution  -  sometimes  called 
'precision  errors'.  If  any  systematic  (or  bias)  errors  exist,  then  their  effect 
would  be  superimposed  on  top  of  the  precision  errors. 


DISTRIBUTION  AFTER  20  POINTS 

theoretical  limits  theoretical 
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Fig  A3  Statistics  of  an  experimental  sample  from  a  Gaussian  distribution 
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RELATIVE  FREQUENCY  DENSITY 


Appendix  A 


41 


NOTATION  USED  IN  APPENDIX  A 


Symbol 

*0 

A1  ' 

*2. 

G 

n 

P 

R 

RELCF 

RELIP 

RSD 

s 

t 

y 

A 

y 

2 


Description 

coefficients  of  least  squares  curve  fit 

Gaussian  random  number 
number  of  values  in  a  sample 

probability,  ie  relative  frequency  of  an  occurrence 

rectangular  random  number 

random  error  limit  of  curve  fit 

random  error  limit  of  individual  points 

residual  standard  deviation 

experimental  standard  deviation 

Student's  statistic  (at  95%  probability  in  this  Appendix) 
ordinate  value  of  a  two-dimensional  point  (vertical  axis) 
curve  fit  value  of  y 

abscissa  value  of  a  two-dimensional  point  (horizontal  axis) 


o  theoretical  standard  deviation  (true  value) 

U  theoretical  mean  value  (true  value) 


No .  Author 

A1  R.W.  Hamming 


REFERENCE  IN  APPENDIX  A 
Title,  etc 

Numerical  methods  for  scientists  and  engineers. 
McGraw  Hill,  2nd  edition  (1973) 


i 


Probability  ol  Refecting  -  Percent 
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OUTLIER  DETECTION 
B .  1  Outliers  from  a  maan  7«lua 

Two  methods  can  be  recommended,  (a)  due  to  Thompson81  and,  (b)  due  to 
Grubbs82.  The  former  is  more  active  in  that  it  will  catch  more  real  wild  points, 
but  it  is  more  prone  to  reject  good  points.  The  latter  will  reject  few  good 
points,  but  will  not  catch  as  many  bad  points.  These  accept/re ject  properties 
were  confirmed  by  Abernethy  using  Monte  Carlo  simulation,  giving  the  results 
shown  in  the  following  two  figures.  Hence  Grubbs'  method  can  safely  be  left  to 
automatic  computer  operation,  but  Thompson's  method  should  only  be  used  to  'flag' 
possible  outliers  for  further  investigation. 


2.5  3.0  3.5  4.0  4.5  5.0 

Outlier  Location 

Number  ol  Standard  Deviations  From  The  Average 


n  mu* 


Fig  Bl  a,P  error  in  Thompson 
outlier  test  (based  on 
one  outlier  in  each  of 
100  samples  of  sizes  5 
10  and  40) 


2.5  3.0  3.5  4.0  4.5  5.0 

Outlier  Location 

Number  ol  Standard  Deviations  From  The  Average 


* 0  '**U* 


Fig  B2  a,p  error  in  Grubbs 

outlier  test  (based  on 
one  outlier  in  each  of 
100  samples  of  sizes  5, 
10  and  40) 
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B.1.1  Thompson's  Tsu  Test 

From  a  sample  of  n  values  of  x  ,  calculate  the  mean  value  x  and  the 
sample  standard  deviation 


(B-l ) 


Note  that  the  divisor  here  is  n  ,  rather  than  (n  -  1) .  For  any  suspected 
outlier,  Xj  calculate 


Tj 


(B-2) 


The  point  is  an  outlier  if  Tj  is  greater  than  the  critical  value  in 
Table  1. 

As  an  example,  take  the  n  -  20  points  of  data  in  Fig  A3  of  Appendix  A, 
where  we  might  suspect  that  point  number  4  is  an  outlier.  Thus,  (using  the 
symbol  'x'  instead  of  ’ G 1 ) 


x4  -  2. 307 
x  =  0.148 


s*  -  0.844 


*4 


12,307  -  0. 1481 
0.844 


2.558  . 


From  Table  Bl,  the  critical  value  is  1.934  for  n  -  20  at  the  5%  level  of 
significance,  hence  this  point  would  be  flagged  as  a  possible  outlier.  It  is 
interesting  to  note  that,  in  fact,  ALL  the  points  in  Fig  A3  are  genuine  values 
from  a  Gaussian  distribution  and  so,  in  this  example,  Thompson's  Tau  Test  has 
been  overactive  in  rejecting  a  good  point!  This  illustrates  the  sound  advice 
that  this  test  should  only  be  used  to  flag  suspect  points  for  further 
investigation . 

B.1.2  Grubbs'  T  Test 

From  a  sample  cf  x  data,  calculate  the  mean  value  x  and  the  standard 
deviation 


-"j 
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<  B— 3 ) 


Note  that  this  time  the  divisor  is  the  usual  (n  -  1)  rather  than  n  .  For 
any  suspected  outlier,  Xj  ,  calculate 


(B-4) 


The  point  is  an  outlier  if  T-j  is  greater  than  the  critical  value  in 
Table  2. 

Taking  the  same  example  from  the  data  in  Fig  A3,  with  point  number  4 
suspected  as  an  outlier,  we  have 


x<  -  2.307 
x  -  0.148 


From  Table  B2,  the  critical  value  is  2.56  for  n  ■=  20  at  the  51 
confidence  level,  hence  the  point  is  correctly  NOT  classified  as  an  outlier. 
Thus  Grubbs'  test  could  safely  have  been  left  to  automatic  computer  operation. 

B .  2  Outliers  from  a  curve  fit 

Outlier  detection  methods  are  not  so  well  established  for  curve  fits  as 
for  mean  values.  However,  there  will  usually  be  a  computer  VDU  graph  available 
for  inspection,  such  as  Fig  7  in  the  main  text,  to  help  the  judgement.  The 
current  practice  at  Pyestock  for  examining  such  scatter  is  explained  first,  in 
section  B.2.1.  An  alternative  method  for  possible  future  use  is  discussed  in 
section  B.2.2. 

B.2.1  Current  practice  at  Pyestock  for  checking  outliers  from  curve 
fits 


The  computer  program  which  produces  the  curve  fits  such  as  Fig  7  of  the 
main  text,  will  draw  lines  of  ± RELCF  and  iRELIP  on  each  side  of  the  curve, 
where 
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±t9SSjp  -  ±[(t953r)2  +  (t95Sy)  2  ] 


(B— 6 ) 


where  3 


and 


fxTy;  -  y)2 
r  ”  V  n  -  p  -  1  ' 


is  the  residual  standard  deviation 


is  the  highest  power  of  the  polynomial 
is  the  standard  deviation  of  the  curve  fit. 


Dealing  first  with  a  straight  line,  we  would  have 


3r  \l+  ,<«!■-  1 

L  n  Kxi  -  x)2  -I 


(B-7) 


Removing  the  t95  factor  from  equation  <B— 6 ) 

s:p  - 


(Equation  { B— 9 )  agrees  with  a  standard  formula  in  Ref  B3 . ) 

With  the  current  outlier  detection  practice  at  Pyestock,  we  would  expect  1 
in  20  points  to  fall  just  outside  the  ± RELIP  boundaries.  For  example,  point 
number  4  in  the  curve  fit  shown  in  Fig  A4  of  Appendix  A  would  be  accepted  as  a 
good  point.  Any  point  which  fell  much  further  away  from  the  curve  would  be 
suspected  as  an  outlier  and  hence  a  candidate  for  deletion.  If  deleted,  the 
curve  fit  would  be  recalculated  with  one  less  data  point,  or  it  might  be  possible 
to  take  another  point  to  keep  the  numbers  up. 

B.2.2  A  possible  alternative  outlier  test  for  curve  fits 

Although  the  current  practice  for  outlier  detection  at  Pyestock  is  as 
described  in  section  B.2.1,  various  alternative  methods  are  becoming  available. 

A  good  candidate  for  possible  future  use,  due  to  Tietjens  et  alB<,  was  one  of  a 
collection  recommended  by  Abernethy  and  Ringhiser—.  This  is  now  described,  with 
a  comment  on  a  tricky  piece  of  philosophy  that  it  contains.  Tietjens  considers 
the  theoretical  model  for  a  point  (x.,y.)  deviating  from  a  straight  line 


(B-S ) 


Sr  [  !  1 
n  X(x  -  x) 2  J 


(B-S) 
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which  is  the  same  as  Tietjens'  equation,  equation  (B-iS) . 
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So#  it  would  seem  that  Tietjens  takes  s*  to  be  the  'pure'  standard 
deviation  of  the  residual  errors,  whereas  Sjp  in  equation  (B— 9 )  of  the  Pyestock 
method  is  the  predicted  standard  deviation  of  the  scattered  points  as  augmented 
by  the  uncertainty  of  the  curve  fit. 

To  proceed  with  Tietjens,  the  ratio  of  the  maximum  deviation  to  its 
standard  deviation  is  found 


R 


(B-22) 


This  is  compared  with  critical  values  in  Tables  B3 .  II  R  exceeds  the  critical 
value,  that  point  is  declared  to  be  an  outlier,  deleted,  and  then  the  curve  fit 
is  recalculated. 

The  above  theory  can  be  extended  to  higher  orders  of  polynomial  curve 
fits.  Thus  from  equation  (B-18) 


Si 


(B-23 ) 


RSDJ  - 

(RELCr)*] 

L 

'  t95  •  J 

All  the  values  for  equation  (24)  are  available  from  the  Pyestock  computer 
program  for  polynomial  curves  up  to  the  cubic. 

The  ratio  for  the  maximum  deviation 


R 


max 


Si 


(B-22)  bis 


is  then  compared  with  critical  values  in  Tables  B4,  taken  from  Ref  B6. 

This  procedure  is  illustrated  by  the  example  of  the  quadratic  curve  fit  in 
Fig  A4  of  Appendix  A.  Of  the  20  points  scattered  about  the  curve,  point  number  4 
is  a  suspect  outlier.  For  this  point  we  have 
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ei 


2.3 


S4 


[  0.952 


/  0 . 65 
'2.11 


>’] 


0.90 


therefore 


R 


2 . 3 
0.9 


2.55  . 


From  Table  B4  the  critical  values  are 


a 

0.10 

0.05 

0.01 

R 

2 . 62 

_ 1 

?  11 

1 _ 

3.06 

and  so  the  suspect  point  number  4  is  declared  'not  guilty',  even  at  the  10% 
level . 

This  verdict  agrees  with  that  of  the  current  Pyestock  method  which,  by 
inspection  of  points  on  Fig  A4,  would  accept  1  point  in  20  lying  just  outside  the 
±  t9ss  limits  of  points  (also  called  ±  RELIP) . 
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Tables  B1  and  B2 


TESTING 

FOR  OUTLIERS 

PROM  MEAN  VALUES 

(Extracted  from 

Ref 

Bl) 

(Extracted  from  Ref  B2) 

Table  1 

Table  2 

Thompson ' s 

Tau 

Rejection  values  for 
Grubbs'  Method 

Table  2.  Rejection  Values  for  Grubbs' 
Table  I.  Thompson's  Tau  Method 


Semple 

Site 

Level  of  Significance 

Sample 

Level  of  Significance 

N 

P-  10% 

5% 

N 

p  -  57 

l  P  -  2.5%  P 

-  ir o 

3 

1.3968 

1.4099 

1.41362 

1.414039 

4 

1.669 

16080 

1.6974 

1.7147 

5 

1.611 

1.767 

1.969 

1.9175 

4 

1.46 

1.48 

1.49 

5 

1.67 

1.71 

1.75 

6 

1.631 

1.814 

1.973 

10609 

6 

1.82 

1.89 

1.94 

7 

1.640 

1.848 

2.040 

1142 

7 

1.94 

2.02 

110 

8 

1.644 

1.870 

2.087 

2207 

8 

103 

2.13 

0  ?? 

9 

1.647 

1.885 

2.121 

1256 

9 

111 

2.21 

132 

10 

1.648 

1.896 

2.146 

2294 

10 

11 

2.18 

123 

129 

136 

141 

148 

12 

129 

141 

155 

11 

1.648 

1.904 

2.166 

2.324 

13 

133 

146 

161 

12 

1.649 

1.910 

2.183 

2.348 

14 

137 

151 

2.66 

13 

1.649 

1.9IS 

2.196 

2.368 

15 

141 

2.55 

171 

M 

1.649 

1.919 

2.207 

2385 

16 

144 

2J9 

175 

IS 

1.649 

1.923 

2.218 

2,399 

17 

147 

2.62 

179 

18 

150 

165 

182 

16 

1.649 

1.926 

2.224 

2.411 

19 

2.53 

168 

185 

17 

1.649 

1.928 

2.231 

2.422 

20 

156 

2.71 

188 

18 

1.649 

1.931 

7  7  37 

1432 

21 

2.58 

173 

191 

19 

1.649 

1.932 

2.242 

1440 

22 

160 

2.76 

194 

20 

1.649 

1.934 

2J47 

1447 

23 

162 

178 

2.96 

24 

164 

180 

199 

21 

1.649 

1.938 

2.251 

1464 

25 

166 

182 

3.01 

22 

1.649 

1.937 

2.255 

1460 

30 

175 

191 

23 

1.649 

1-938 

2^59 

2.466 

35 

2.82 

2.98 

24 

1.649 

1.940 

2-262 

1470 

40 

187 

3.04 

23 

1.649 

1.941 

2.264 

1475 

45 

192 

3.09 

50 

196 

3.13 

26 

1.848 

1.942 

2.267 

1479 

60 

3.03 

3.20 

27 

1.648 

1.942 

2-269 

1483 

70 

3.09 

326 

28 

1.648 

1.943 

2.272 

1487 

80 

3.14 

3.31 

29 

1.648 

1.944 

2.274 

1490 

90 

3.16 

3.35 

30 

1.648 

1.944 

1275 

1493 

100 

3.21 

3.38 

31 

1.648 

1.946 

1277 

2.495 

32 

1.648 

1.946 

1279 

1498 

00 

1.64485 

1.95996 

132634 

157582 

Appendix  B 


Table  B4  (concluded) 


(fl  -  >05) 


1 _ 9 _ 

n 

I 

2 

4 

5 

6 

d 

1  o 

1  5 

26 

5 

1  .  V  2 

6 

2.07 

1  .43 

7 

2.  14 

2.  rji 

1.94 

M 

2  •  2-* 

2.20 

2.  10 

1 .94 

9 

2.35 

2.2-; 

2.21 

2.  10 

1.95 

j  n 
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12 
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14 
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2.  Sri 

2.55 
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16 

2  •  66 

2.66 
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2.60 
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2.43 

2.26 

U 
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2.72 

2.70 
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2.65 

2.62 

2.55 

2.44 

20 
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2.77 

2.76 

C.  74 

2.72 
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25 
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2. 67 

2.46 

2  *  44 

2.C3 

2.33 

2.76 

2.60 

30 

2.96 

2.46 

2.95 
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2.90 

2.4ri 

2.79 

2.  17 

3l> 

3.03 

3.02 

3.02 

3.01 

3.00 

3.00 

2.93 

2.97 

2.9  1 
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40 

3.33 

3.  Ori 

3.07 

3.07 

3.06 

3.06 

3.05 

3.03 

3.03 

2.34 

45 

3.13 

3.  12 

3.  12 

3.  12 

3.11 

3.  1 1 

3.  10 

3.39 

3.06 

2.96 

53 

3.  17 

3.16 

3.  16 

3.  16 

3.15 

3.  15 

3.  14 

3.  14 

3.  1  1 

3.04 

63 

3.23 

3.23 

3.23 

3.23 

3.22 

3.22 

3.22 

3.21 

3.20 

3.15 

70 

3.23 

3.24 

3.2ri 

3.2ri 

3. 2d 

3.  2ri 

3.  27 
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-JO 

3.33 

3.33 
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3.33 

3.33 

3.33 

3.32 

3.  32 
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3.37 
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3.34 

3.34 
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3.41 

3.4  7 
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3.40 

3.40 
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3.39 

3. 3d 
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“1 
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» 

2 

3 

u 

5 

h 
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IS 

5 

1  .95 

6 
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7 
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t 
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2.18 
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9 
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10 
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2.  18 

t  .99 

12 
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2.64 
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14 
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2.  d  2 
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20 
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25 
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3.16 

3.14 
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30 
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3. 2d 
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35 
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3.33 

3.33 
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3.  19 

2.81 

40 

3.43 

3.42 

3.42 

3.41 
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3.40 

3.  Sri 

3.36 

3.30 
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45 
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3.47 

3.47 
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3.23 

50 

3.52 
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3.51 

3.51 

3.51 

3.50 

3.49 

3.43 

"3.45 

3.34 

60 

3.60 

3.59 

3.59 

3.59 

3.58 

3.5d 

3.57 

3.56 

3.54 
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70 
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3.64 
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3.63 

3.61 
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3.70 

3.70 
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3.67 
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90 
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3.74 

3.74 

3.73 
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3.72 

3.70 

100 

13.78 

3.76 

3.78 

3.77 

3.77 

3.77 

3.77 

3.77 

3.76 

3.74 

n  *  nuraber  of  observation* 


q  ■  nurr.ber  of  independent  variables  (including  count  for  intercept  if 
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NOTATION  USED  IN  APPENDIX  B 


Roman  symbols 


Description 


bi 

e 

n 

P 

q 

R 

RELCF 

RELIP 

RSD 

s 

s* 

Si 

Sip 

Sr 
3  * 

y 

T 

x 

*i 

X 

y 

*i 


first  coefficient  in  linear  regression 
second  coefficient  in  linear  regression 
vertical  deviation  of  point  from  regression  line 
number  of  points 

highest  power  in  a  polynomial  equation 

number  of  coefficients  (including  the  intercept) 
in  s  p^vnomial  equation  (Table  4) 

max  ratio  of  a  residual  to  its  standard  deviation 

random  error  limit  of  curve  fit 

random  error  limit  of  individual  point 

residual  standard  deviation 

Grubbs'  standard  deviation,  with  (n  -  1)  divisor 

Thompson's  standard  deviation,  with  n  divisor 

'pure'  standard  deviation  of  residuals,  used  by  Tietjens 

standard  deviation  of  individual  point 

residual  standard  deviation 

standard  deviation  of  curve  fit 

Student’s  statistic  at  95%  confidence 
Grubb's  statistic 

either:  a  one-dimensional  random  variable 

or:  a  value  of  two-dimensional  data  on  the  horizontal  axis 
an  individual  value  of  x 
a  suspected  outlier  of  x-data 
mean  value  of  x-data 

a  value  of  two-dimensional  data  on  the  vertical  axis 
an  individual  value  of  y 

a  curve-fit  value  of  y 


Greek  symbols 
a 

P 

P: 

Pi 

e 

6: 

X 


either:  type  of  error  due  to  incorrect  rejection 
or:  level  of  significance  in  critical  tables 

type  of  error  due  to  incorrect  acceptance 

first  regression  coefficient  in  theoretical  model 

second  regression  coefficient  ir.  theoretical  model 

theoretical  Gaussian  error  in  regression  model 

residual  standard  deviation 

Thompson's  tau  statistic 
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Notes : 

1  Much  of  the  notation  in  this  Appendix  is  that  of  the  original 
references . 

2  The  symbols  crr  ,  RSD  ,  Sr  are  synonymous. 
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No .  Author 

B1  W.R.  Thompson 


B2  F.E.  Grubbs 


B3  P.G.  Moore 

D.E.  Edwards 
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GLOSSARY 


Note:  This  Glossary  is  intended  for  quick  reference  with  enough  explanation  to 
act  as  a  convenient  reminder  of  the  main  features  of  the  methodology. 

Accuracy  A  conversational  type  of  word  used  to  convey  a  qualitative 

impression  of  the  goodness  of  measurements  and  results. 
However,  the  word  is  so  popular  and  has  so  many  different 
meanings,  that  it  is  now  too  confusing  to  use  in  a  technical 
sense . 


Actual 

Results 

Analysis 


Average 


Basic 

Measurement, 

xi 


Bias  (or 
Systematic) 
Error,  P 

Bias  Limits, 
+B 

Curve  Fitting 


The  statistical  analysis  of  the  random  scatter  about  the  test 
result,  usually  by  analysis  of  a  performance  curve  to  calculate 
its  Residual  Standard  Deviation  (RSD)  and  Random  Error  Limit  of 
Curve  Fit  (RELCF) .  This  involves  only  the  precision  errors 
which  appear  with  the  test  result.  Bias  errors,  being 
constant,  are  not  revealed. 

Synonymous  with  'mean  value'.  The  random  uncertainty  of  an 
average  of  'n'  points  is  1  /•v/n  times  the  uncertainty  of  a 
single  point,  ie  it  is  thereby  improved. 

(Sometimes  called  an  'Input  Parameter'.) 

One  of  a  number  of  variables  (such  as  airmeter  static  pressure, 
PSA,  or  fuel  flow  rate,  QF,  etc)  which  appear  in  the 
mathematical  expressions  through  which  a  te3t  result  is 
calculated  from  the  measurements.  A  separate  Elemental  Error 
Table  is  published  for  each  Basic  Measurement  -  see  Table  1  for 
an  example. 

A  constant  error  which  lies  somewhere  within  the  limits  ±B  . 
The  value  of  the  error  itself  is  unknown  but  the  limits  can  be 
estimated  by  'Prediction  Synthesis'. 

The  maximum  plus/minus  values  within  which  a  bias  error  is 
expected  to  lie. 

Synonymous  with  'regression',  a  process  by  which  a  best  curve, 
of  some  pre-defined  mathematical  form,  is  fitted  to  the  test 
data  by  the  method  of  least  squares.  The  statistical 
properties  of  a  curve  fit  are  similar  to  those  of  a  mean  value. 


Curve  Shift 
Effect 


Defined 

Measurement 

Process 


Defined 

Test 

Result 


Degrees  of 
Freedom,  V 


Elemental 

Errors 


Error 

Categories,  j 


GLOSSARY  (continued) 

Because  engine  performance  results  are  displayed  as  graphical 
correlations  against  other  results,  it  is  necessary  to  account 
for  the  propagation  of  errors  not  only  directly  to  the  vertical 
ordinate,  y  ,  but  also  indirectly  via  the  horizontal  abscissa, 
z  .  Bias  errors  in  z  do  not  affect  the  y-value  of  a  point 
but  they  do  shift  the  position  of  the  curve  (see  Fig  17) . 

This  embraces  every  factor  and  every  measurement  that  has  an 
influence  upon  the  'Defined  Test  Result',  not  only  in  the  test 
itself,  but  also  stretching  all  the  way  back  through  the 
calibration  hierarchy  to  the  National  Standards. 

In  ordinary  usage,  the  'Test  Result'  would  be  an  engine 
performance  parameter  (such  as  Net  Thrust,  FN  or  Specific  Fuel 
Consumption,  SFC,  etc)  the  determination  of  which  is  the  object 
of  the  test.  In  this  document,  the  'Defined  Test  Result'  has 
an  extra  special  meaning  for  the  purpose  of  uncertainty 
assessment.  For  this  it  is  necessary  to  declare  exactly  what 
form  the  result  is  to  be: 

either  a  single  point  value,  R 

or  a  mean  value,  R 

or  a  curve  fit  value,  R  . 

The  number  of  independent  points  of  data  involved  in  a  statis¬ 
tical  calculation.  For  a  mean  value,  \>  -  n  -  1  .  For  a  curve 
fit  u-n-p-1  ,  where  p  is  the  power  of  the  polynomial 
curve . 

These  are  the  most  fundamental  errors  that  can  affect  each 
Basic  Measurement.  They  are  accounted  as  various  Error  Sources 
in  four  Error  Categories. 

For  each  Basic  Measurement,  there  are  four  categories  into 
which  it  is  convenient  to  group  the  various  Error  Sources  for 
accounting  purposes,  thus 


(1) 

Calibration  hierarchy 

(2) 

Data 

acquisition 

(3) 

Data 

reduction 
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Error 

Classification 


Error 

Propagation 


Error 

Sources 

Fossilisation 


GLOSSARY  (continued) 

It  is  first  necessary  to  declare  exactly  what  is  meant  by  the 
'Defined  Test  Result',  for  example  an  SFC  curve.  Then  any 
error  which  produces  random  scatter  in  that  result  is 
classified  as  'Precision',  and  any  error  which  remains  constant 
(and  hence  unseen)  in  that  result  is  classified  as  'Bias'. 

The  Bias  Limits  of  each  Elemental  Error  within  each  Error 
Category,  j  ,  are  first  assessed  and  accounted  in  a  separate 
table  for  each  Basic  Measurement,  x^  .  These  are  combined 
separately  by  root-sum-squares,  thus 


and  recorded  at  the  bottom  of  the  separate  tables  (eg  see 
Table  1) .  These  Bi  values  for  each  Basic  Measurement,  xi  , 
are  then  propagated  to  the  Test  Result,  Rk  ,  by  means  of  the 
appropriate  Influence  Coefficients,  and  again  combined  by  root- 
sum-squares,  thus 


B, 


a/FF^TF 


to  give  the  total  effect  on  each  test  result  (eg  see  Table  2) . 

In  principle,  similar  prediction  synthesis  could  be  applied  to 
the  Precision  Indices  but,  in  RAE  practice  at  Pyestock,  it  is 
considered  better  to  assess  this  by  Actual  Results  Analysis 
(qv)  . 

These  are  the  most  fundamental  origins  to  which  the  Elemental 
Errors  can  be  ascribed,  within  each  Error  Category. 

Random  errors  which  occur  before  the  start  of  the  engine  test 
run,  for  example  during  an  instrument  calibration,  will  leave  a 
small  fixed  error  in  the  calibration  curve.  This  fixed  error 
is  then  propagated  to  the  engine  test.  Thus  such  effects  of 
random  errors  are  'fossilised'  into  bias  errors  in  the  engine 
test  result. 
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Gaussian 

Distribution 


Influence 

Coefficient, 


(The  alternative  name  is  ‘Normal  Distribution'.) 

The  bell-shaped  probability  distribution  of  random  variables, 
which  is  described  in  all  text  books  of  statistics.  The  most 
likely  value  is  the  mean  value,  while  95%  of  values  are  found 
within  ±2ct  ,  where  a  is  the  theoretical  standard  deviation. 
In  practical  terms,  c  is  replaced  by  the  Precision  Index, 
s  ,  or  by  the  Residual  Standard  Deviation,  RSD  . 

The  rate  of  change  of  a  Test  Result,  Rk  ,  with  respect  to  a 
Basic  Measurement,  x^ 


ie  0ik 


dRk 

dxj 


or  0£k 


Rk  dxj 


Precision 
(or  Random) 
Error,  t 


Precision 
Index,  s 


Prediction 

Synthesis 


in  relative,  or  percentage  form. 

A  random  variable,  assumed  to  come  from  a  Gaussian  distribu¬ 
tion,  which  lies  somewhere  within  the  limits  its,  or  itRSD. 

The  presence  of  such  errors  is  shown  by  the  scatter  of  points 
about  a  mean  value,  or  about  a  curve  fit.  The  value  of  's'  or 
'RSD'  is  found  by  Actual  Results  Analysis,  and  Student’s  "t"  is 
extracted  from  statistical  tables. 

The  experimental  standard  deviation  of  the  Gaussian  distribu¬ 
tion  from  which  the  Precision  Errors  arise.  In  principle,  the 
value  of  's'  could  be  estimated  by  Prediction  Synthesis  but  it 
is  RAE(P)  practice  to  calculate  it  from  statistical  analysis  of 
actual  data. 

This  is  important  as  the  only  way  to  assess  the  effects  of  bras 
errors.  (Precision  errors  can  also  be  included  but  these  are 
best  left  to  actual  results  analysis.)  Every  error  source  in 
the  defined  measurement  process  is  examined  and  assessed  for 
elemental  bias  limits,  with  a  separate  table  for  each  basic 
measurement  (or  input  parameter!  .  These  bias  limits  are  ther. 
propagated  to  the  test  result  by  means  of  influence 
coefficients,  and  combined  to  predict  the  overall  effect. 


lira 
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Probability 

Probability 

Distribution 


RELCF 


RELIP 

Residual 
Standard 
Deviation,  RSD 

Student's  "t" 


GLOSSARY  (continued) 

The  likelihood  of  a  value  being  realised.  The  word 
’probability'  is  synonymous  with  'relative  frequency'. 

In  diagrammatic  form,  the  horizontal  axis  shows  all  possible 
values  of  a  variable  (x  ,  say)  while  the  vertical  axis  shows 
the  'probability  density'  of  x  ,  denoted  as  dP/dx  (eg 
Figs  A1  and  A2  of  Appendix  A) .  The  probability  of  a  value  of 
x  occurring  between  A  and  B  is  given  by 


dP 


ie  the  area  under  the  curve  between  A  and  B  .  The  total 
area  under  the  curve 


J 


dP 

dx 


dx 


is  equal  to  1,  ie  total  probability. 

The  random  error  limits  of  curve  fit  which  describe  a  band 
enclosed  by  the  limits  ± RELCF  on  either  side  of  a  curve  fit. 

If  there  are  no  bias  errors,  the  true  curve  will  be  expected  t 
lie  somewhere  within  this  band,  at  95%  confidence.  The 
statistical  properties  of  RELCF  are  similar  to  those  of  a  mean 
value,  ie  the  band  ±RELCF  is  narrower  than  the  scatter  of 
individual  points. 

The  random  error  limits  of  individual  points  scattered  about  a 
curve  fit.  These  limits  are  approximated  by  ±2RSD. 

The  standard  deviation  of  the  residual  scatter  of  points  about 
a  curve  fit.  It  is  calculated  by  statistical  methods  ir.  Actua 
Results  Analysis. 

A  factor,  extracted  from  statistical  tables  against  the 
appropriate  degrees  of  freedom,  to  give  the  confidence  limits 
its,  or  itRSD,  within  which  a  precision  error  is  expected  tc 
lie.  The  confidence  level  is  usually  taken  to  be  95%. 
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Uncertainty 


GLOSSARY  (concluded) 

This  word  is  used  instead  of  'Accuracy'  in  a  technical  treat¬ 
ment,  such  as  the  present  document.  Its  numerical  value  is  the 
half-range  of  an  interval  within  which  the  error  is  expected  to 
lie.  Uncertainty  has  two  components  of  error,  bias  and 
precision,  which  may  be  combined  in  alternative  ways.  Thus, 
for  the  Uncertainty  of  a  single  point, 

Uadd  -  B  +  2s 
or 

Urss  -  V B2  +  (2s)2  . 

(Note:  The  Precision  Index,  s  ,  could  be  replaced  by  the  RSD.) 
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Roman  symbols 
B 

BLCF 

CF 

CL95 

DMP 

FNRD 

G 

n 

NHRD 

P 

P7Q2 

P 

RELCF 

RELIP 

RSD 

s 

SFCRD 

TV 

t 

Uadd 

Urss 

UETP 

Ur 

WA1RD 

x 

y 


NOTATION 

(see  Glossary  for  detailed  explanations) 

Description 

bias  limit 

bias  limit  of  curve  fit 
curve  fit 

95%  confidence  limit 

defined  measurement  process 

net  thrust,  referred  to  desired  conditions 

a  number  from  a  Gaussian  distribution 

number  of  points 

high  pressure  shaft  speed,  referred  to  desired  conditions 
probability  (ie  relative  frequency) 
engine  pressure  ratio  -  P7/P2 
either  (a)  a  test  result 

or  (b)  a  number  from  a  random  rectangular  distribution 
random  error  limit  of  curve  fit 
random  error  limit  of  individual  points 
residual  standard  deviation 

precision  index  (ie  experimental  sample  standard  deviation) 
specific  fuel  consumption,  referred  to  desired  conditions 
true  value 

Student's  "t"  factor,  usually  for  95%  confidence 
additive  uncertainty  «  (B  +  ts) 
root-sum-square  uncertainty  -  \B2  +  (ts)2 
uniform  engine  test  programme 
random  uncertainty  ■  ts 

engine  inlet  air  flow  rate,  referred  to  desired  conditions 
a  basic  measurement 

a  main  test  result,  plotted  against  graph  ordinate 
a  secondary  test  result,  plotted  against  graph  abscissa 
(ie  a  correlating  result) 


65 


NOTATION  (concluded) 


Greek  symbols 


Description 


P 

e 


a 


a  bias  (systematic)  error 
a  precision  (random)  error 

influence  coefficient  of  a  result  Rk  to  a  basic  measurement,  xi 
dRk 

dxi 

theoretical  population  standard  deviation 


Superscripts 

-  mean  value 

*  curve  fit  value 


Suffixes 

i 

j 

k 


general  basic  measurement 
general  source  of  elemental  error 
general  test  result 
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